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An  Advanced  Distributed  Simulation  Inclusive  Test  Planning  Methodology 

a  paper  by  Mr.  John  M.  Reeves, 

SAIC  Program  Manager  for  JADS  Technical  Support 

The  methodology  described  in  this  paper  is  couched  in  terms  of  operational  test  and  evaluation 
(OT&E).  But,  as  OT&E  moves  left  on  the  acquisition  timeline  and  as  new  systems  demand  ever 
more  complex  test  environments,  the  process  can  benefit  Development  Test  and  Evaluation 
(DT&E)  as  well. 

Historically,  OT&E  planners  have  had  to  live  with  severe  test  asset  and  cost  limitations.  As  a 
result,  the  tendency  during  test  concept  development  has  been  to  analyze  the  operating 
environment  from  the  bottom  up.  Typically,  the  process  involved  identifying  the  set  of  entities  in 
the  operating  environment  which  has  direct  interaction  with  the  system  under  test  and  culling  that 
group  down  to  a  minimum  set.  The  historical  approach  resulted  in  a  consistent  collection  of  test 
limitations  which  are  found  in  test  report  after  test  report,  e.g.,  insufficient  numbers  and  types  of 
targets;  insufficient  numbers  and  types  of  friendly  players;  and  inadequate  representation  of 
command,  control,  communications,  intelligence,  surveillance,  and  reconnaissance  assets  on 
friendly  and  opposing  sides. 

The  advantage,  at  the  cost  of  test  limitations,  of  the  historical  approach  to  test  concept 
development  and  planning  was  that  uncontrolled  variables  were  kept  to  a  minimum,  and  cause  and 
effect  relationships  were  relatively  straightforward.  That  was  not  a  trivial  advantage,  but  it  was 
an  advantage  for  the  analysts  not  the  system  under  test  (SUT)  users.  In  combat  operations  the 
users  sometimes  found  that  factors  not  included  in  operational  testing  had  significant  bearing  on 
the  ability  of  a  system  to  do  its  intended  job. 

It  is  a  plausible  working  assumption  that  advanced  distributed  simulation  (ADS)  can  technically 
support  representation  of  the  military  operating  environment  at  the  campaign  or  theater  level.  If 
ADS  is  included  in  the  test  planning  tool  kit  from  the  outset,  it  is  possible  to  begin  the  test 
concept  development  process  at  the  top  rather  than  the  bottom.  (The  “top”  may  not  be  at  theater 
level;  it  is  established  by  the  relevant  operational  task.)  The  methodology  described  in  this  paper  is 
a  top-level  methodology.  It  is  an  approach  which  is  compatible  with  the  “strategy  to  task” 
philosophy.  It  is  also  a  methodology  for  test  planning  which  incorporates  the  consideration  of 
ADS  —  it  is  not  an  ADS  planning  methodology.  This  methodology  is  designed  to  provide 
insights  on  whether  to  use  ADS  and  where  in  a  test  program  the  use  might  fit. 

The  advantage  of  a  top-down  approach  to  test  concept  development  is  that  the  whole  gamut  of 
interactions  is  available  for  consideration  even  if  many  of  those  interactions  are  assessed  as 
irrelevant  and  excluded  from  the  final  concept.  The  top-down  approach  doesn’t  require  that 
every  possible  interaction  be  included  in  the  test,  but  it  does  require  an  item  by  item  assessment  of 
each  interaction.  Decisions  to  exclude  interactions  are  conscious  decisions  not  default  decisions 
as  a  function  of  a  bottom-up  approach. 

Mission  or  task  level  evaluation  is  explicitly  a  top-down  approach.  The  top  level,  for  test 
planning  purposes,  may  be  much  lower  than  campaign  or  theater.  Just  how  high  the  top  level  is, 
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is  a  function  of  the  task  being  evaluated.  Some  systems  may  have  little  or  no  interaction  beyond  a 
unit  boundary,  and  others  may  interact  closely  with  the  theater  and  campaign  levels.  In  the  case 
of  DT&E,  it  may  make  sense  to  substitute  “specification  sets”  for  “tasks”.  The  methodology 
should  still  be  useful,  but  the  cast  of  players  for  most  DT&E  would  be  small  relative  to  the  OT&E 
cases. 

NOTE:  A  top-down  approach  won’t  help  much  if  it  is  implemented  with  a  mind  set  fixed  on 
historical  limitations.  It’s  necessary  for  the  test  planners  to  understand  that  ADS  provides 
opportunities  which  weren’t  previously  feasible. 

In  order  to  lend  structure  to  the  methodology,  I  have  elected  to  assume  a  SUT  which  is  a 
hypothetical  air-ground  attack  aircraft.  Please  bear  in  mind  that  this  is  just  an  example  of  a 
process.  Some  of  the  terminology  may  be  dated  or  inaccurate,  but  I  trust  the  logic  for  the  process 
will  hold  up.  I  believe  the  reader  will  have  little  difficulty  tailoring  the  steps  of  the  process  to 
update  the  terminology  or  to  apply  them  to  other  kinds  of  systems.  The  logic  flow  for  the  initial 
elements  of  the  planning  methodology  is  shown  in  Figure  1. 

Step  1.  Understanding  the  System  Under  Test.  Step  1  requires  the  test  planners  to  research 
the  acquisition  documentation  to  gain  a  thorough  understanding  of  the  SUT  and  its  intended 
operating  environment.  This  understanding  incorporates  the  operational  tasks  the  system  is 
designed  to  perform,  the  critical  system  parameters,  the  system  and  operational  requirements,  the 
concept  of  operations,  the  logistical  support  concept,  and  the  top  level  or  general  operating 
environment.  One  piece  of  the  understanding  deals  with  the  technical  or  specification  aspects  of 
the  system.  The  other  piece  deals  with  the  interactions  between  the  technical  characteristics  of 
the  system  and  the  world  it  operates  in  from  a  strategic  perspective  —  the  friendly  and  supporting 
forces,  the  natural  environment,  and  the  threats  posed  by  the  enemy.  In  the  case  of  a  hypothetical 
close  air  support  (CAS)  aircraft,  the  cast  of  potential  players  from  a  theater  perspective  is  pretty 
extensive: 

•  Joint  force  headquarters 

•  Air  operations  centers  (AOC) 

•  Air  task  orders 

•  Fragmentary  orders 

•  Wing  operations  centers  (WOC) 

•  Tactical  operations  centers  (corps,  division,  brigade,  etc.) 

•  Theater  air  defense  structures 

•  Air  defense  units 

•  Control  centers 

•  Safe  passage  procedures 

•  Tactical  air  control  parties 

•  Forward  air  control  posts 

•  Airborne  battlefield  command  and  control  centers  (ABCCC) 

•  Forward  air  controllers-air  (FAC- A) 

•  Forward  observers  (FO) 
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•  Air  and  naval  gunfire  liaison  companies  (ANGLICO) 

•  Direct/indirect  support  units  (suppressive  fires) 

•  Logistical  support  agencies/units 

•  Munitions 

•  Consumables 

•  Air  support  assets 

•  Combat  Air  Patrol 

•  Escort 

•  Suppression 

•  Refueling 

•  Combat  search  and  rescue 

•  Airborne  Warning  And  Control  System  (AWACS) 

•  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS) 

•  Airborne  laser  (ABL) 

•  Rivet  Joint,  Guardrail,  and  other  stuff 

•  Allied  forces 

•  Threats 

•  Airborne  interceptors 

•  Surface-to-air  missiles  (SAM) 

•  Anti-aircraft  artillery  (AAA) 

•  Small  arms 

•  Ground  forces 

•  Airborne 

•  Special  forces 

•  Naval  forces 

•  At  sea 

•  Ashore 

•  Targets 

•  Troop  units 

•  Vehicles 

•  Fixed  targets 

•  Weapon  emplacements 

Obviously  not  all  of  the  listed  players  play  in  every  task;  as  the  task  changes,  the  relevant  players 
list  changes. 

Technical  measures  are  straightforward  and  map  directly  to  system  requirements  and 
specifications.  The  technical  measures  tend  to  remain  constant  through  the  range  of  operational 
tasks,  although  some  of  them  may  be  stressed  in  some  tasks  and  not  in  others. 

Operational  measures  map  to  the  ability  of  a  system  to  perform  its  intended  tasks.  The 
environment  and  the  player  set  may  change  significantly  between  tasks  (e.g.,  day  to  night  or 
interdiction  to  close  air  support).  The  operational  measures  may  vary  significantly  from  task  to 
task. 
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Once  the  test  planners  have  a  thorough  understanding  of  the  system,  its  tasks,  its  operating 
environment  and  its  interactions  in  that  environment,  they  can  proceed  to  the  next  step. 

Step  2.  Select  a  Task.  This  step  involves  the  selection  of  a  specific  task.  A  complex  system  may 
be  assigned  many  operational  tasks.  Some  tasks  may  be  very  similar,  while  others  may  be  vastly 
different.  It  is  possible  that  similar  tasks  may  be  grouped  for  evaluation  purposes  and  tested  on 
the  basis  of  a  single  task.  In  our  hypothetical  case  we  might  choose  a  task  of  CAS  in  daylight  and 
good  weather  with  communications  jamming  (comjam),  and  a  moderate  threat  environment. 

Step  3.  Develop  Relevant  Measures.  Once  a  specific  task  is  selected,  the  planners  can  develop 
relevant  measures  for  the  task  and  a  task-specific  operating  environment.  The  operating 
environment  in  combination  with  assigned  objectives  and  missions  provides  a  context  for  the  test 
measures  and  defines  the  cast  of  players.  Let’s  assume  for  our  illustrative  case  that  we  have 
measures  associated  with  the  following  areas  of  SUT  performance: 

•  Ability  of  the  pilot  in  the  CAS  aircraft  to  communicate  effectively  with  controlling, 
coordinating  and  supporting  agencies  in  a  comjam  environment 

•  Survivability 

•  Ability  to  navigate  to,  identify,  and  attack  the  appropriate  target 

•  Circular  error  probable  (CEP),  circular  error  average  (CEA) 

•  Weapons  effects 

Given  those  kinds  of  measures,  the  relevant  task  player  list  might  look  something  like  the 
following: 

•  System  under  test  (SUT) 

•  Air  operations  center  (AOC) 

•  Air  task  order 

•  Fragmentary  order 

•  Wing  operations  center  (WOC) 

•  Corps  tactical  operations  center 

•  Theater  air  defense  structure 

•  Control  center 

•  Safe  passage  procedure 

•  Airborne  battlefield  command  and  control  centers  (ABCCC) 

•  Airborne  forward  air  controller  (FAC) 

•  Forward  observers  (FO) 

•  Air  support  assets 

•  Suppression 
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•  Refueling 

•  Airborne  Warning  And  Control  System  (AWACS) 

•  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS) 

•  Allied  forces 

•  Threats 

•  Airborne  interceptors 

•  Surface-to-air  missiles  (SAM) 

•  Anti-aircraft  artillery  (AAA) 

•  Small  arms 

•  Targets 

•  Troop  units 

•  Vehicles 

In  order  to  structure  a  test,  the  player  cast  has  to  be  embedded  in  a  dynamic  operational  scenario. 
The  scenario  supports  detailed  mission  layout  activities  and  time-sequenced  events  for  the  SUT. 
Some  of  the  players  on  the  list  have  very  tightly  coupled  interactions  with  the  SUT.  Examples 
include  friendly  ground  forces,  enemy  ground  forces  (targets),  terminal  threats,  and  the  airborne 
FAC.  Other  agencies  such  as  AWACS,  Joint  STARS,  or  the  WOC  have  more  loosely  coupled 
interactions  but  may  be  important  to  SUT  performance. 

Step  3  is  finished  when  the  cast  of  players  has  been  whittled  down  to  those  who  play  a  role  in  the 
performance  of  the  SUT  in  the  selected  task,  and  the  measures  of  performance  (MOP)  and 
measures  of  effectiveness  (MOE)  have  been  developed  for  the  SUT  evaluation.  The  scenario 
developed  in  Step  3  is  an  operational  scenario  —  not  a  test  scenario.  That  comes  later. 

Step  4.  Consider  Using  ADS.  The  activity  described  to  this  point  is  simply  a  test  planning  or 
test  concept  development  approach.  Step  4  is  a  switch  point  —  to  include  or  not  to  include  a 
detailed  consideration  of  ADS  use  as  part  of  the  planning  methodology.  In  a  few  cases,  the 
operating  and  test  environments  may  be  relatively  simple.  An  example  might  be  the  testing  of  a 
new  side  arm.  A  test  of  such  a  system  can  be  done  live,  in  open  air,  and  affordably.  The  example 
I  have  chosen  is  complex  and  highly  interactive  with  a  wide  cast  of  players.  In  the  complex  case, 
it  is  reasonably  clear  that  live,  open  air  testing  with  the  full  cast  of  players  is  not  affordable. 
Because  of  the  large  number  of  man-in-the-loop  (MITL)  interactions,  it  is  also  reasonably  clear 
that  representation  of  the  environment  in  a  stand-alone  simulation  will  suffer  credibility  problems 
because  we  don’t  model  human  decision  making  very  well.  If  the  test  planners  are  reasonably 
certain  that  the  test  environment  cannot  be  adequately  represented  using  the  traditional  test 
approaches,  then  they  have  two  choices:  accept  the  test  limitations  or  explore  ADS  to  see  if  the 
technology  can  make  a  better  test  within  the  fiscal  constraints. 

Stated  succinctly,  the  decision  associated  with  Step  4  is  about  the  adequacy  of  the  test  scenario  as 
compared  with  the  operational  scenario.  In  a  world  with  no  fiscal  or  safety  constraints,  the 
operational  scenario  and  the  test  scenario  would  be  identical.  In  the  real  world,  the  issue  becomes 
“do  we  have  to  approximate  reality  to  have  a  satisfactory  test.”  If  the  test  planners  cannot 
provide  an  appropriate  test  environment  using  traditional  test  approaches,  then  the  answer  is  “no,” 
and  the  planners  should  explore  whether  ADS  can  do  anything  to  improve  the  situation.  If  the 
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answer  is  “no,”  then  the  process  moves  along  to  step  5.  If  the  answer  is  “yes,”  then  the  planners 
can  proceed  with  a  traditional  test  plan  for  that  particular  operational  task.  Other  tasks  may 
require  different  approaches. 

Step  5.  Select  a  Player.  Traditional  testing  shortfalls  often  include  an  insufficient  number  of  test 
articles,  insufficient  number  of  threats,  and  inadequate  representation  of  friendly  force 
interactions.  The  process  of  ADS  exploration  begins  with  a  visit  to  the  player  list  developed  in 
Step  3,  and  the  first  player  on  that  list  is  the  SUT.  Depending  on  where  the  program  is,  the  SUT 
may  be  available  in  a  variety  of  forms.  Early  in  the  program,  the  SUT  may  only  be  available  as  a 
digital  system  model  (DSM).  Later  the  SUT  may  exist  in  brassboard  form  with  a  variety  of 
subcomponents  scattered  among  a  variety  of  vendors.  Eventually  the  SUT  will  be  available  in 
prototype  or  production  version  form,  and  a  flight  simulator  version  will  emerge.  (The  DSM 
version  is  still  available  at  this  stage.) 

Step  6.  Other  Players.  Let’s  assume  that  we  have  a  prototype  version  of  the  SUT  and  a  flight 
simulator  exists.  We  also  have  a  DSM  form  of  the  SUT.  If  the  operational  doctrine  calls  for  a 
two-ship  flight,  then  we  have  a  requirement  to  represent  two  SUT.  One  approach  could  be  to 
couple  the  SUT  manned  flight  simulator  with  another  manned  simulator  that  could  represent  the 
SUT  and  the  communications  types  and  channels  for  interflight  communications.  A  facility  like 
the  Tactical  Air  Command  and  Control  Simulation  Facility  (TACOSF)  at  Kirtland  Air  Force  Base, 
New  Mexico,  could  represent  a  second  version  of  the  SUT  at  a  reasonable  cost.  Most  manned 
dome  simulators  could  provide  the  visual  representation  of  the  TACCSF  generated  wingman. 
The  technology  exists  to  input  or  overlay  jamming  energy  on  the  communications  link. 

After  looking  at  the  SUT  configuration  choices,  the  planners  can  move  on  to  the  other  players.  If 
the  player  list  is  prioritized  on  the  basis  of  the  more  direct  interactions,  then  the  more  important 
players  are  addressed  first.  For  each  player,  the  test  planners  must  have  access  to  information 
about  the  various  manifestations  of  the  player.  They  must  know  what  forms  are  available,  and 
they  must  learn  what  they  can  about  capabilities  and  costs  for  each  form. 

A  facility  like  TACSSF  could  represent  the  airborne  FAC  complete  with  man-in-the-loop.  If  there 
was  a  higher  fidelity  requirement,  a  manned  simulator  of  the  FAC  aircraft  could  be  used.  The 
communications  links  could  be  subjected  to  jamming  in  the  same  manner  as  the  interflight  links. 

The  targets  and  the  friendly  ground  forces  in  the  vicinity  of  the  strike  also  have  tight  interactions 
with  the  SUT.  Individual  targets  can  be  simulated  in  large  numbers  by  a  number  of  constructive 
simulations  —  Janus  serves  as  an  example.  Target  visual  representations  can  be  piped  to  the 
simulator  dome  for  the  SUT.  Friendly  command  and  control  centers  of  all  types  can  be 
represented  at  a  level  of  fidelity  tailored  to  test  needs  and  cost.  A  corps  tactical  operations  center 
(TOC),  for  instance,  could  be  represented  by  a  full-up,  fielded  TOC  if  testing  was  done  in 
conjunction  with  some  exercise.  The  TOC  could  also  be  represented,  at  decreasing  levels  of 
fidelity  and  cost,  as  a  partially  manned  TOC  in  garrison;  as  an  emulation  of  a  TOC  in  TACCSF  or 
a  similar  simulation  facility;  or  by  a  single  individual  reading  from  a  script.  The  same  approach  can 
be  applied  to  other  command  and  control  elements  such  as  the  tactical  air  control  party. 


6 


Another  tight  interaction  involves  the  SUT  and  the  threat  systems  in  the  strike  area.  The  threats 
can  be  represented  at  any  level  from  constructive  to  live.  If  high  fidelity  is  a  requirement  (and  in 
our  hypothetical  example  it  is)  and  since  survivability  is  a  measure  of  interest,  then  hardware-in- 
the-loop  may  be  a  minimum  requirement. 

As  the  players  (the  AOC  and  the  WOC,  for  instance  )  become  more  and  more  peripheral,  fidelity 
requirements  decrease  and  scripted  inputs  or  constructive  models  will  suffice. 

Step  6  involves  a  lot  of  research  and  learning. 

Step  7.  Fidelity  Versus  Cost.  Step  7  involves  the  art  of  compromise  between  fidelity  and  cost. 
With  the  information  gleaned  in  Step  6,  the  test  planners  are  in  a  position  to  make  reasoned 
choices  about  the  players  in  the  test  and  the  appropriate  form  of  representation  for  each  of  them. 

Step  8.  Evaluate  Adequacy  of  the  Environment.  When  the  process  reaches  Step  8,  the 
question  facing  the  planners  involves  the  adequacy  of  the  environment  which  will  be  created  by 
the  interactions  of  the  players  selected.  Each  time  steps  5  through  7  are  executed,  the  planners 
must  ask  whether  more  players  are  needed.  If  so,  they  return  to  Step  5.  Step  5  is  executed 
repeatedly  until  the  test  planners  are  satisfied  that  the  test  environment  is  rich  enough  in  terms  of 
meaningful  interactions  to  support  a  sound  test. 

Step  9.  Detailed  Planning.  When  the  test  planners  are  satisfied  they  have  a  sound  test 
framework  for  a  task  oriented  test,  they  can  press  on  with  detailed  planning  to  include 
networking,  instrumentation,  data  collection  systems,  scheduling,  etc. 

Step  10.  Additional  Tasks.  Step  10  can  be  performed  sequentially  after  each  iteration  of  Step  9 
is  completed,  or  if  there  are  enough  planning  resources,  planning  for  additional  tasks  can  go  on  in 
parallel.  In  any  case,  test  planning  is  not  complete  until  all  of  the  relevant  operational  tasks  have 
been  addressed. 

Step  11.  Develop  Master  Plan.  The  task  associated  with  step  1 1  involves  the  deconfliction  and 
coordination  of  each  of  the  individual  task  oriented  test  segments.  Essentially  step  1 1  involves  the 
development  of  the  master  plan  and  schedule. 

In  summary,  this  is  a  top-level  planning  process  which  incorporates  decision  elements  associated 
with  ADS  for  test  and  evaluation.  It  is  only  intended  as  an  example,  and  each  test  program  is 
going  to  have  unique  aspects  which  will  require  planners  to  use  creativity  and  imagination. 
Hopefully  this  outline  approach  provides  the  test  planners  with  an  understanding  that  ADS  offers 
test  opportunities  that  were  never  there  before.  ADS  can  be  used  to  good  effect  at  reasonable 
cost  if  it  is  used  intelligently.  There  are  many  things  ADS  won’t  remedy,  but  there  are  many 
things  it  will.  ADS  is  a  technology  set  which  should  be  considered  during  test  planning.  If  it 
doesn’t  make  sense  to  use  it,  don’t  use  it.  On  the  other  hand,  don’t  dismiss  it  just  because  you 
think  it’s  too  hard,  or  too  exotic,  or  too  expensive. 
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Acronyms  and  Definitions 


AAA 

anti-aircraft  artillery 

ABCCC 

airborne  battlefield  command  and  control  center 

ABL 

airborne  laser 

ADS 

advanced  distributed  simulation 

ANGLICO 

air  and  naval  gunfire  liaison  companies 

AOC 

air  operations  center 

AWACS 

Airborne  Warning  and  Control  System 

CAS 

close  air  support 

CEA 

circular  error  average 

CEP 

circular  error  probability 

COI 

critical  operational  issue 

comjam 

communications  jamming 

CONOPS 

concept  of  operations 

DSM 

digital  system  model 

DT&E 

developmental  test  and  evaluation 

FAC 

forward  air  controller 

FO 

forward  observer 

Janus 

interactive,  computer-based  simulation  of  combat  operations 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

MITL 

man-in-the-loop 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

ORD 

operational  requirements  document 

OT&E 

operational  test  and  evaluation 

RCM 

Requirements  Correlation  Matrix 

SAM 

surface-to-air  missile 

STAR 

Surveillance  Target  Attack  Radar  System 

SUT 

system  under  test 

T&E 

test  and  evaluation 

TACCSF 

Tactical  Air  Command  and  Control  Simulation  Facility,  Kirtland  AFB, 
New  Mexico 

TOC 

tactical  operations  center 

woe 

wing  operations  center 
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